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A  methodology  for  modeling  and  evaluating  measures  of  effectiveness  of 
Cy  systems  is  described.  The  approach  combines  the  framework  provided  by 
the  Modular  Command  and  Control  Evaluation  Structure  (MCES)  and  the 
quantitative  methods  of  the  System  Effectiveness  Analysis  approach.  The 
application  to  a  C*  problem  in  which  a  test  bed  is  used  is  outlined. 
Finally,  the  question  of  credibility  is  addressed,  when  only  data  from  test 
beds  and  demonstrations  are  available  for  evaluating  measures  of 
effectiveness . 
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1 .  INTRODUCTION 


Improvements  in  weapon  system  technology,  and  higher  capacity  and 
speed  in  data  transmission,  combined  with  an  increasing  complexity  of  the 
battlefield,  impose  severe  time  constraints  on  hardware,  software,  and 
human  decisionmakers .  The  purpose  of  this  paper  is  to  present  a 
methodology  that  is  being  developed  at  MIT  for  modeling  and  computing 
measures  of  performance  (MOPs)  and  measures  of  effectiveness  (MOEs)  for  C* 
systems.  The  methodological  framework  was  first  applied  to  C1  systems  by 
Bouthonnier  and  Levis  [1]  and  then  extended  in  a  series  of  theses  and 
papers:  Cothier  and  Levis  [2],  Karam  [3],  Karam  and  Levis  [4],  Bohner  [5], 
and  Martin  [6] . 

In  the  last  several  years,  a  series  of  workshops  have  been  held  at  the 
Naval  Postgraduate  School  in  an  effort  to  develop  generic  tools  for 
evaluating  C*  systems  and  architectures  [71.  The  Modular  Command  and 
Control  Evaluation  Structure,  or  MCES,  provides  a  framework  for  evaluating 
C*  architectures  in  which  many  different  paradigms,  models,  and  algorithmic 
procedures  can  be  embedded.  In  this  paper,  an  effort  is  made  to  show  how 
the  MCES  framework  and  the  quantitative  System  Effectiveness  Analysis 
approach  can  be  integrated  to  provide  a  useful  way  for  defining  and 
evaluating  MOEs  for  C*  systems. 

This  Integration  has  become  possible  because  both  approaches  are  based 
on  the  same  six  concepts:  system,  environment,  context,  parameters, 
measures  of  performance,  and  measures  of  effectiveness.  The  first  three 
are  used  to  pose  the  problem,  while  the  last  three  define  the  key 
quantities  in  the  analytical  formulation  of  the  problem.  The  analytical 
aspects  of  the  methodology  mainly  address  how  hardware  characteristics, 
system  structure,  and  standard  operating  procedures  related  to  system 
performance . 

The  system  consists  of  components,  their  interconnections,  and  a  set 
of  operating  procedures.  A  boundary  can  be  drawn  that  defines  what  is 
included  within  the  system  whose  effectiveness  is  to  be  assessed;  what  is 


included  depends  on  the  analysis  at  hand.  The  environment  consists  of  our 
own  forces  and  the  adversary's  forces,  upon  which  our  forces  can  act  and 
which  can  act  upon  ours.  For  example,  a  C*  system  is  used  to  monitor 
(sense)  the  environment  and  to  direct  forces.  Engagements  between  two 
forces  in  an  urban  area  or  at  a  mountain  pass  define  typical  environments . 
The  context  denotes  the  set  of  conditions  and  assumptions  within  which  the 
system  and  the  environment  exist.  The  relationship  between  system, 
environment,  and  context  is  shown  in  Figure  1. 


Figure  1.  System,  Environment,  and  Context 

Parameters  are  the  independent  quantities  used  to  specify  the  system  and 
the  mission  requirements.  For  example,  in  the  case  of  a  fire  support 
system,  system  parameters  may  include  quantities  that  describe  the 
detection  equipment,  computational  time  delays,  kill  radius  of  the 
munition,  and  failure  probabilities  associated  with  the  components,  to  name 
but  a  few.  Parameters  of  the  mission  may  be  the  tempo  of  operations,  as 
described  by  the  speed  of  the  threats,  and  the  size  of  the  engagement. 

Measures  of  Performance  are  quantities  that  describe  system  properties  or 
mission  requirements.  MOPs  for  a  command  and  control  system  may  include 


reliability,  survivability,  cost,  and  probability  of  lclll.  The  aission 
requirements  should  be  expressed  by  the  same  quantities  as  the  system  MOPs, 
e.g.,  minimum  reliability  or  survivability,  maximum  cost,  or  minimum 
probability  of  kill.  System  parameters  are  defined  within  the  system 
boundary;  MOPs  may  be  defined  within  the  boundary,  or  they  may  Include 
aspects  of  the  environment. 

Measures  of  Effectiveness  (MOEs)  are  quantities  that  result  from  the 
comparison  of  the  system  MOPs  to  the  mission  requirements.  They  reflect 
the  extent  to  which  the  system  meets  the  requirements.  To  evaluate  the 
MOEs,  it  is  necessary  to  go  outside  the  boundary  and  consider  the 
environment .  Only  then  could  the  effect  of  the  system  on  the  mission 
outcome  be  evaluated. 

In  this  methodology  for  assessing  effectiveness,  the  system  MOPs  and 
the  mission  requirements  must  be  modeled  and  analyzed  independently,  but  in 
a  conmon  context.  The  system  capabilities  should  be  determined 
independently  of  the  mission  requirements,  and  the  mission  requirements 
should  be  derived  without  considering  the  system  to  be  assessed. 
Developing  requirements  with  a  specific  system  implementation  in  mind  leads 
to  obvious  problems  regarding  the  credibility  of  the  assessment. 

The  analytical  formulation  of  the  methodology  for  system  effectiveness 
analysis  (SEA)  and  its  relationship  to  the  Modular  Command  and  Control 
Evaluation  Structure  (MCES)  are  described  in  Section  2,  while  a  recent 
application  to  a  problem  is  described  briefly  in  Section  3.  Finally,  in 
Section  4  some  general  consideration  on  measures  of  effectiveness  are 
presented . 

2.  THE  TWO  APPROACHES:  MCES  AND  SEA 

The  Modular  Command  and  Control  Evaluation  Structure  (MCES)  is  shown 
in  Figure  2.  It  consists  of  seven  modules  organized  in  a  sequential 
manner.  In  reality,  an  analyst  employing  MCES  would  iterate  between  the 


modules,  as  appropriate,  until  a  satisfactory  solution  to  the  problem  was 
obtained . 


Figure  2.  Modular  Command  and  Control  Evaluation  Structure  (MCES) 

In  the  first  module,  the  decisionmaker's  requirements  are  expressed  in 
the  form  of  a  problem  statement  consisting  of  a  set  of  objectives  and  the 
associated  assumptions.  In  the  second  module,  the  problem  statement  is 
used  to  bound  the  problem,  i.e.,  specify  the  boundaries  of  the  system  to  be 
analyzed.  This  is  the  most  critical  part  of  the  MCES  methodology  and  the 
one  that  often  requires  a  number  of  iterations.  The  result  is  the 


identification  of  the  system  components  and  their  interconnections.  Zn  the 
third  module,  the  particular  command  and  control  process  is  described.  The 
result  is  the  specification  of  the  set  of  functions  such  as  "sense*, 
"assess",  "generate",  "select",  and  "direot"  [8],  The  allocation  of  the 
functions  derived  in  module  3  to  the  components  and  structure  is  carried 
out  in  module  4.  Thus,  in  the  first  four  steps,  the  complete  formulation 
of  the  problem  is  achieved. 

The  next  three  modules  constitute  the  "solution”  to  the  problem.  In 
module  S,  the  various  measures  that  are  relevant  for  the  problem  in 
question  are  specified:  MOPs,  MOEs  and,  if  appropriate.  Measures  of  Force 
Effectiveness  or  MOFEs.  Such  measures  as  survivability,  reliability,  and 
interoperability  are  typical  examples  of  MOPs.  However,  these  measures 
represent  general  concepts)  there  is  need  for  problem-specific  variables 
that  are  measurable  and  can  represent  these  MOPs,  i.e.,  instantiations  for 
the  specific  C*  system  being  evaluated.  The  values  of  these  variables 
should  be  computable  from  data  generated  by  the  system.  Finally,  in  module 
7.  the  aggregation  of  MOEs  is  carried  out. 

The  MCES  methodology,  as  described  briefly  above,  provides  a  logical 
and  orderly  structure  that  guides  the  analyst  through  the  process  of 
formulating  the  measures  of  effectiveness  that  are  appropriate  for  the 
problem  in  question.  The  System  Effectiveness  Analysis,  (SEA)  however, 
focuses  on  the  quantitative  aspects  of  obtaining  and  evaluating  measures  of 
effectiveness.  Indeed,  the  steps  of  the  SEA  can  be  embedded  in  MCES  and 
especially  in  the  last  three  modules . 

The  first  step  in  (SEA)  consists  of  defining  the  System,  the 
Environment,  and  the  Context,  followed  by  the  selection  of  the  parameters 
that  influence  the  system  MOPs.  By  definition,  these  parameters  are 
considered  mutually  independent,  since  they  constitute  the  "independent 
variables”  in  the  analytical  formulation  of  the  methodology.  This  step  is 
a  specifc  implementation  of  Modules  1  to  4  in  MCES. 
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In  the  second  step,  the  analogous  procedure  is  carried  out  for  the 
mission.  Parameters  of  the  mission  are  defined  that  are  consistent  with 
the  environment  and  the  context.  This  step  does  not  have  a  direct 
correspondence  to  the  modules  of  MCE S,  but  is  implicit  in  the  feedback  loop 
between  the  aggregation  modules  and  the  decisionmaker  (Fig.  2). 


The  third  step  consists  of  defining  MOPs  for  the  system  that 
characterize  the  properties  that  are  of  interest  in  the  analysis.  The  MOPs 
are  expressed  as  functions  of  the  parameters.  The  values  of  the  MOPs  could 
be  obtained  from  the  evaluation  of  a  function,  from  a  model,  from  a 
computer  simulation,  from  a  test  bed,  or  from  empirical  data.  Each  MOP 
depends,  in  general,  on  a  subset  of  the  parameters,  i.e.. 
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MOPs  may  or  may  not  be  independent  from  each  other/  two  measures  are 
interdependent,  if  they  have  parameters  in  common.  A  system  realization 
results  in  the  set  of  parameters  taking  specific  values  (x1).  Substitution 
of  these  values  in  the  relationships  (1)  yields  values  for  the  set  {MOP}. 
Thus,  any  specific  realization  can  be  depicted  by  a  point  in  the  MOP  space. 

The  fourth  step  consists  of  selecting  the  models  that  map  the  mission 
parameters  y^  into  the  requirements : 


m  * 


Some  of  the  mission  requirements  may  be  interrelated  through  dependence  on 
common  parameters.  It  is  also  possible  to  introduce  directly  some 
constraints  between  the  requirements,  e.g.,  a  trade-off  relationship 
between  delay  and  accuracy.  However,  it  is  preferable  that  such  trade-off 
relationships  be  derived  through  the  functions  or  models  that  define  MOPs 
or  requirements  in  terms  of  the  mission  parameters.  Specification  of 
values  for  the  mission  parameters  results  in  a  point  or  region  in  the 
mission  requirements  spaoe . 
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The  two  spaces,  the  system  MOP  space  and  the  mission  requirements 
space,  although  of  the  same  dimension,  may  be  defined  in  terms  of  different 
quantities,  or  quantities  scaled  differently.  Therefore,  the  fifth  step 
consists  of  transforming  the  system  measures  and  mission  requirements  into 
a  set  of  commensurate  attributes  defined  on  a  common  attribute  space.  For 
example,  one  of  the  system  MOPs  may  be  vulnerability,  while  the 
corresponding  mission  requirement  may  be  survivability.  Since  they  both 
reflect  the  same  concept  -  the  effect  of  external  actions  -  one  of  them  may 
be  chosen  as  the  common  attribute,  say,  survivability,  while  the  other  one 
will  then  be  mapped  into  the  first  one. 

The  Measures  of  Performance  for  the  system  are  functions  of  the  system 
parameters.  Consequently,  as  the  x's  in  Eq.  (1)  vary  over  their  allowable 
ranges,  the  MOPs  take  values  that  generate  a  locus  in  the  MOP  space.  This 
transformation  is  shown  in  Figure  3.  The  resulting  locus  is  called  the 
System  Locus,  L_. 
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Analogously,  the  set  of  values  that  satisfy  the  mission  requirements, 
Eq.  (2),  constitute  the  Mission  Locus,  Lm  (Fig.  4).  The  two  loci  are 
constructed  in  step  6,  after  the  common  MOP  space  has  been  defined  in  step 
5. 
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Pigure  4.  Mission  Locus 


The  seventh  step  is  the  key  one  in  analyzing  the  effectiveness  of  a  C* 
system  in  view  of  the  mission  that  is  to  be  carried  out.  It  consists  of 
procedures  for  comparing  the  system's  MOPs  and  mission's  requirements 
through  the  geometric  properties  of  two  loci.  The  geometric  relationship 
between  the  two  loci  can  take  one  of  three  forms: 


(1)  The  two  loci  do  not  have  any  points  in  common,  i.e.,  the  intersection 
of  L3  and  1^  is  null: 


l  n  l 

s  m 


In  this  case,  the  system  does  not  satisfy  the  mission's  requirements,  and 
one  would  define  the  effectiveness  to  be  zero,  regardless  of  which  specific 
measure  is  used. 

(2)  The  two  loci  have  points  in  common,  but  neither  locus  is  included  in 
the  other: 


L„  n  L  *  *. 
s  m 


(4) 


L  n  L  <  L  and  L 
sms  m 


(5) 


In  this  case,  a  subset  of  the  values  that  the  MOPs  may  take  satisfies  the 
mission  requirements.  Many  different  measures  can  be  used  to  described  the 
extent  to  which  the  system  meets  the  requirements.  Each  of  these  measures 
may  be  considered  an  MOE.  For  example,  let  V  be  such  a  measure.  Then  an 
effectiveness  measure  can  be  defined  by 

E  =  V(L  O  L  )/V(L  )  (6) 

1  sms 

which  emphasizes  how  well  the  system  capabilities  are  used  in  the  mission, 
while 


E  -  V(L  n  L  )/V(L  )  (7) 

*  s  m  m 

expresses  the  degree  of  coverage  of  the  requirements  by  the  system 
capabilities . 

(3)  The  mission  locus  is  included  in  the  system  locus: 


L  n  L 


s  m 


Lm  * 


(8) 


In  this  case,  it  follows  from  (8)  that  Lg  is  larger  then  Lm  and, 
consequently,  the  ratio  defined  by  (6)  will  be  less  than  unity.  This 
result  can  be  interpreted  in  two  ways.  First,  only  certain  system 
attribute  values  meet  the  requirements  of  the  mission.  The  second 
interpretation  is  that  the  use  of  this  system  for  the  given  mission 
represents  an  inefficient  use  of  resources  since  the  system  capabilities 
exceed  the  mission  requirements.  Inefficiency,  in  turn,  implies  lower 
effectiveness . 
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The  measures  of  effectiveness  given  by  (6)  or  (8)  are  partial 
measures.  Let  these  partial  measures  be  denoted  by  {Er}.  To  combine  these 
into  a  single  global  measure,  utility  theory  may  be  used.  Therefore,  the 
subjective  judgements  of  the  system  designers  and  the  users  cam  be 
incorporatred  directly  into  the  methodology  in  two  ways:  (1)  by  choosing 
different  partial  measures,  and  (2)  by  selecting  a  utility  function.  The 
global  effectiveness  measure  is  obtained,  finally,  from 

E  =  u(Ei,Ei....,Ek).  (9) 

This  is  the  last  step  of  the  SEA  methodology  and  corresponds  to  the  seventh 
module  of  the  MCES. 

3.  EFFECTIVENESS  OF  AN  AIR  DEFENSE  SYSTEM* 

In  the  1970's,  the  problem  of  aircraft  identification  in  the  NATO 
environment  was  investigated  and  the  implication  of  indirect  or  third  party 
ID  was  explored.  Indeed,  because  of  the  speed  of  the  threats  and  the  short 
reaction  time  available,  indirect  ID  for  distinguishing  friends,  foes,  and 
neutrals  (IFFN)  becomes  crucial. 

To  assess  the  effectiveness  of  the  indirect  ID  process,  a  test  bed  was 
developed  in  order  "to  assess  baseline  capabilities  within  the  air  defense 
C*  structure  to  perform  the  IFFN  function,  to  identify  deficiencies  in  the 
performance  of  that  function,  and  to  define  potential  near-term  procedural 
and  equipment  modifications  for  further  testing".  This  test  bed  [9]  is  a 
surrogate  for  the  actual  system  which,  of  course,  can  hardly  be  tested.  By 
conducting  experiments  on  the  IFFN  Joint  Test  and  Evaluation  facilities,  it 
is  expected  that  the  effectiveness  of  the  actual  system  can  be  assessed. 


•This  section  is  based  on  the  recent  thesis  by  Philippe  J.  F.  Martin  [6] 


The  IFFN  System  considered  is  shown  in  Figure  5;  it  is  composed  of 
nodes  corresponding  to  Fire  Direction  Centers  (FDC) ,  Control  Reporting 
Centers  (CRC) ,  Special  Information  System  (SIS)  and  the  weapons  systems 
(Patriot,  Hawk,  F-15,  F-16) . 


Figure  5.  Structure  of  the  Actual  IFFN  System 


The  mission  of  the  system  is  to  engage  and  destroy  hostile  airborne 
targets  and,  therefore,  deny  the  enemy  access  to  the  defended  airspace  and 
to  prevent  attacks  on  friendly  assets.  At  the  same  time,  the  system  must 
prevent  attacks  on  friendly  or  neutral  aircraft.  To  accomplish  this 
mission,  the  air  defense  system  must  perform  a  number  of  functions: 
detection,  tracking,  identification,  allocation,  engagement,  and  target 
kill. 

As  a  first  order  approximation  to  the  actual  problem,  a  simplified 
model  was  introduced  [6]  for  which  five  system  parameters  were  defined: 


(1)  time  needed  to  transmit  information  from  one  node  to  another  (it 
depends  on  whether  the  SIS  is  included  or  not) ; 

(2)  range  from  aircraft  to  the  Fire  Support  Coordination  Line  (FSCL)  at 
the  time  of  detection;  it  depends  on  the  variable  Air  Control 
Procedure . 

(3)  quality  of  identification; 

(4)  level  of  centralization  of  control;  and 

(5)  quality  of  target  allocation  and  engagement.  This  is  dependent  on  the 
quality  of  the  Q  and  A  IFF  devices. 


The  MOPs  for  assessing  the  system  should  have  clear  physical  meaning 
—  that  helps  in  defining  variables  for  measuring  them  —  and  reflect  the 
tasks  to  be  performed.  Three  MOPs  have  been  defined. 

The  first  MOP  reflects  the  success  of  the  air  defense  task;  it  is 
measured  by  the  normalized  ratios  of  the  remaining  friendly  forces  to  the 
remaining  enemy  forces.  The  second  MOP  measures  the  number  of  neutrals 
shot  down  by  the  air-defense  system.  The  last  MOP  is  the  distance  from  the 
FSCL  that  the  enemy  has  achieved  when  the  air  battle  ends. 

When  the  five  parameters  are  allowed  to  vary  over  their  admissible 
ranges,  i.e.. 


pi,min  Pi  “  pi,max 


i  =  1 . 5 


then  the  complete  system  locus  is  obtained.  The  plots  of  the  locus  have 
been  generated  by  a  special  purpose  graphics  package  [5]  for  an  IBM  PC/AT 
and  an  HP  7475  plotter.  In  Figures  6  and  7,  the  two  dimensional 
projections  of  the  locus  Ls  are  shown. 

The  mission  requirements  have  been  assumed  to  be  the  following:  that 
the  friendly  forces  'win*  the  battle,  that  a  small  percentage  of  neutrals 
is  engaged,  and  that  the  enemy  forces  have  not  been  allowed  to  come  within 
range  of  friendly  assets  on  the  ground.  These  conditions  define  a 
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rectangular  parallilepiped  in  the  MOP  space  (Figure  8) . 

Mission  locus 


Figure  8.  Projections  of  the  Mission  Locus 

The  appropriate  measure  of  effectiveness  is  the  one  given  by  Eq.  (6). 
For  the  hypothetical  values  used  in  this  example,  Ex  was  calculated  to  be 
approximately  0.3.  The  actual  value  is  not  significant;  what  is 
significant  is  that  the  methodology  makes  it  possible  to  compute  changes  in 
the  value  when  functions  or  nodes  are  eliminated  from  (or  added  to)  the 
system  [6] . 

4 .  CONCLUSION 

In  the  previous  three  sections,  a  quantitative  methodology  for 
assessing  C*  systems  was  described  and  an  illustrative  example  outlined. 
The  methodology  was  imbedded  in  the  MCES  which  provides  a  framework  for 
formulating  C*  system  evaluation  problems.  The  use  of  the  IFFN  test  bed  as 
the  illustrative  example  raises,  however,  a  new  class  of  C*  issues  that  are 
relevant  for  large  scale  systems  that  are  not  easily  tested  in  the  field 
under  realistic  operational  conditions.  The  most  prominent  of  these  issues 
is  one  of  credibility:  how  credible  are  the  results  obtained  from  models, 
simulations,  and  test  beds  with  respect  to  the  actual  system’s  performance. 
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A  usual  approach  to  the  problem  Is  to  Increase  the  fidelity  of  the 
components  in  the  expectation  that  the  test  bed  will  be  a  "credible* 
surrogate  for  the  real  system.  However,  increased  fidelity  leads  to 
lnoreased  complexity  and.  hence,  reduction  in  the  transparency  of  the 
results . 

In  considering  the  role  of  models  and  test  beds  in  the  assessment  of 
C*  systems,  three  functional  groups  can  be  identified:  (a)  the  modeler,  or 
test  bed  builder,  who  conceives,  designs,  and  implements  the  test  bed;  (b) 
the  user  who  is  concerned  with  obtaining  quantitative  and  qualitative 
information;  and  (c)  tb»  decisionmaker,  who  requires  pertinent,  timely 
analyses  and  recommendation*  from  his  staff. 

If  the  identification  of  the  three  functional  groups  is  accepted,  then 
the  Issues  of  validity,  verification,  and  credibility  can  be  put  in 
perspective. 

Verification  is  a  test  of  whether  the  model  or  test  bed  behaves  exactly  as 
its  designer  intended. 

Validation  is  a  test  of  whether  the  model  or  test  bed  behavior  is  in 
agreement  with  the  real  system  it  represents  with  respect  to  the  specific 
purposes  for  which  the  model  has  been  designed. 

Verification  includes  testing  software,  testing  algorithms  for 
convergence,  checking  that  data  are  entered  properly  and  used  correctly 
and,  generally,  ensuring  that  the  implemented  model  is  true  to  its 
conception  and  error  free.  Verification  does  not  address  the  question  of 
whether  the  conception  is  valid. 

Validation  is  an  issue  between  the  model  user  and  the  modeler  because 
validity  is  closely  tied  to  the  function  the  model  will  perform.  A  model 
may  be  'valid*  for  one  set  of  uses  and  invalid  for  many  others  that  appear 
to  be  very  similar.  While  the  differences  may  seem  minor,  they  may  violate 
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some  of  the  implicit  assumptions  in  the  model 


What  is  important  in  considering  a  "verified"  model  for  use  in 
assessment  is  whether  its  underlying  assumptions  and  theories,  or  the 
approximations  regarding  the  model's  boundaries  and  the  interactions 
between  the  subsystems,  are  compatible  with  the  issue  to  be  analyzed.  Even 
though  a  model  may  have  been  verified  by  the  modeler  and  considered  valid 
by  the  model  user  for  a  specific  study,  there  is  still  a  third  issue: 
credibility  of  the  model  with  decision  makers  [10].  G.  L.  Johnson  has 
stated  that  "problem  solving  analyses  are  credible  with  decision  makers,  if 
they  pass  four  tests:  (1)  coherence,  (2)  correspondence,  (3)  clarity,  (4) 
workability  "  [11].  The  first  two,  coherence  and  correspondence,  can  be 
interpreted  as  tests  for  validation  and  verification,  while  the  clarity 
test  is  one  of  understandability.  "When  the  decision  maker  applies  the 
test  of  workability,  he  checks  the  proposed  solution  and  projected 
consequence  of  the  solution  to  see  if  he  thinks  the  outcomes  would  actually 
result  from  the  prescribed  action  and,  if  they  result,  whether  they  will 
solve  the  problem  before  him*  111],  A  fifth  test  can  be  added  for  evolving 
systems,  that  of  repeatability,  which  measures  the  consistency  over  time 
with  which  a  model  generates  understandable  and  workable  results. 

This  concept  of  credibility,  with  its  attendant  objective  and 
subjective  tests,  forms  interactive  links  between  the  three  entities  -  the 
modeler,  the  user,  and  the  decisionmaker. 

In  view  of  the  expected  increase  in  the  use  of  test  beds  and 
demonstrations,  credibility  may  well  become  a  dominant  measure  of 
effectiveness  in  the  future.  Unfortunately,  there  are  hardly  any 
procedures  or  tools  for  assessing  credibility.  Both  conceptual  and 
analytical  work  is  needed  to  address  this  issue. 
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